The present invention relates to a method for improving the visual appearance of a digital image. In a specific embodiment, the invention relates to reducing staircasing effects as a result of electronic zooming of a low resolution image. The method is suitable for enhancing digital images obtained by an electronic camera.
Original images are usually represented electronically by a rectangular array of picture elements or pixels, which together make up the electronic image. Each pixel represents a small rectangular portion of an original image. The length of each rectangular portion or inversely, the number of pixels per unit of length in the original image is referred to as the spatial resolution of the electronic image. The spatial resolution is expressed in pixels per millimeter. Usually the resolution is constant in each direction, such that each pixel represents a square portion. The electronic image may also have a constant maximum resolution in a first direction and a constant minimum resolution in a second direction orthogonal to the first direction. In that case, each pixel represents a non-square rectangular portion of the original image. The resolution is dictated by the size of the original image to be represented electronically and the maximum number of pixels available in a row or a column of the rectangular array for the electronic image. If the size of the original image is 1 meter and the number of pixels is 1000, then the resolution of the electronic image will be 1000 pixels per meter or 1 pixel per mm. That electronic image may be rendered on a device capable to print 10 pixels per millimeter, i.e. a device having a spatial device resolution of 10 pixels per mm. As such, the 1000 pixels of the electronic image will be rendered on 100 mm. The original image of 1 meter will thus be rendered on 1 decimeter, i.e. at a scale of {fraction (1/10)}. If however the original image must be rendered at a scale of ⅕, i.e. the electronic image must occupy 200 mm, then two options are available. As a first option, a device may be selected having a spatial device resolution of 5 pixels per mm. Then 1000 pixels will be rendered on 200 mm. Alternatively, as a second option, the device having a resolution of 10 pixels per mm may be used, to render a second electronic image of 2000 pixels, derived from the first electronic image having 1000 pixels. The first electronic image has a resolution of 1 pixel per mm, the second electronic image still refers to 1 meter of the original image, but represented by 2000 pixels. As such, the second electronic image has a spatial resolution of 2000 pixels per meter, i.e. 2 pixels per mm. Thus, the spatial resolution of the second electronic image is twice the resolution of the first electronic image. Several techniques are known in the art to convert a first electronic image having a first spatial resolution to a second electronic image having a second spatial resolution. Such techniques are referred to image scaling. The most common and easiest image scaling technique is referred to as pixel replication or nearest neighbour interpolation. To double the spatial resolution of a first electronic image according to pixel replication to obtain a second electronic image having a higher spatial resolution, in a first step each pixel is replicated horizontally in each one image line of the electronic image, i.e. the first pixel of the first electronic image is copied to the first and second pixels of an intermediate electronic image; the second pixel of the first electronic image is copied to the third and fourth pixels of the intermediate electronic image; etc. In a second step, each line of the intermediate electronic image is replicated vertically in the second electronic image, i.e. the first line of the intermediate electronic image is copied to the first and second lines of the second electronic image; the second line of the intermediate electronic image is copied to the third and fourth lines of the second electronic image; etc. By this scaling technique, a single black pixel on a white background in the first electronic image results in a block of 2 by 2 black pixels on a white background in the second electronic image. A black horizontal line having a width of one pixel on a white background in the first electronic image results in a black horizontal line having a width of two pixels on a white background in the second electronic image. The same applies for a vertical line. An oblique black line on a white background in the first low resolution electronic image, e.g. a line having a slope of 45xc2x0 and having a width of 1 pixel in the first image, i.e. a set of single black pixels connected to each other by the upper right and the lower left corners of the pixel, will result in the second high resolution electronic image in a set of black squares of 2xc3x972 pixels each, connected to each other by their upper right and lower left corners. This effect, which is even more conspicuous for oblique lines having slope different from 45xc2x0, is referred to as staircasing. The staircasing effect is also visible in the enlarged image where shorter oblique line segments are present in the original electronic image, or even where an abrupt transition from a white area to a black area is present and the line of transition is neither horizontal nor vertical. The above example describes a scaling factor of 2. Also other integer scaling factors such as 3, 4, etc. may be envisaged. Even non-integer scaling factors are possible, such as a factor of 1.5 where each even pixel of the first electronic image is replicated once and each odd pixel of the first electronic image is replicated twice.
By use of another scaling technique, referred to as bilinear interpolation, transitions are smoothed, in the sense that between a black and a white pixel of a transition, a grey pixel will be introduced by the interpolation technique. Also this may be an unwanted effect, since the originally sharp transitions tend to look fuzzy on the enlarged image.
A third scaling technique, referred to as cubic spline interpolation, creates the new pixels of the second high resolution electronic image by convolution of four pixels of one line in the first low resolution electronic image with a third order polynomial, which:
1. is zero outside the interval x=[xe2x88x922,2],
2. takes the value y=1 in x=0,
3. takes the value y=0 in x=xe2x88x922, xe2x88x921, +1 and +2
4. is everywhere continuous, i.e. for x=(xe2x88x92∞,+∞); and,
5. has everywhere continuous first and second order derivatives.
This convolution technique applied horizontally and vertically substantially eliminates the staircasing effect and the smoothing effect on the grey values, but may introduce other artefacts known to be due to reconstruction, such as overshooting, where an abrupt transition from light grey pixels to dark grey pixels may introduce pixels having a greyness lighter than the light grey or darker than the dark grey.
Despite the above mentioned drawbacks of image scaling, this operation becomes increasingly necessary in electronic image processing applications for the following two reasons. First of all, low cost image scanners or one-shot electronic cameras may produce low resolution images. The size of the original image or scene may be fixed, e.g. 210 mmxc3x97297 mm on an A4 page to be scanned or 5 m horizontally by 3.75 m vertically of an object in the scene to be captured by an electronic camera. Also the number of pixels horizontally and vertically may be fixed, such as 1280 pixels horizontally and 960 pixels vertically as in the ePhoto 1280 (Trademark) electronic camera distributed by Bayer Corporation, Agfa division Wilmington, Mass. As such, the object of the scene will be stored in the electronic image at a resolution of 256 pixels per meter. If the user reproduces the object of the scene by printing the electronic image on a 400 dpi (15.75 pixels per mm) printer, the size of the reproduced image will be:
1280 pixels/15.75 pixels/mm=81 mm horizontally; and,
960 pixels/15.75 pixels/mm=61 mm vertically.
If the user wants the object of the scene to be reproduced on a usual size for photo prints, e.g. 10 cmxc3x9715 cm, the resolution of the captured electronic image may be doubled to obtain a second electronic image of 2560 by 1920 pixels, having a spatial resolution of 512 pixels per meter. If printed on the same 400 dpi printer, the size of the reproduction will be 12.2 cm by 16.2 cm. If a nearest neighbour interpolation technique were used to electronically increase the resolution by a factor of two, staircasing may already be visible by the naked eye. If the user wants to fill an A4 sized page (21 cm by 29.7 cm) by the reproduced scene, the captured low resolution image may be scaled by a factor of four to obtain an image of 24.4 cm by 32.4 cm, which may be cropped to the required size. In such case however, the staircasing effects are clearly visible and even objectionable. These effects are also visible on portions of an electronic image that are electronically enlarged to be viewed in more detail on the colour monitor of an interactive image processing station. In an image acquisition system, simply the number of individual photosensitive elements in a row or column are referred to as the image resolution. This stems from the fact that such number is usually fixed for a specific device, whereas the size of the scene or the original picture which is projected on such row or column may be varied by optical zooming. The ePhoto 1280 described above may thus be said to have a xe2x80x9cresolutionxe2x80x9d of 1280. This xe2x80x9cresolutionxe2x80x9d is imposed by the size of the two-dimensional CCD (charge coupled devices) array in the electronic camera. The question whether or not staircasing will be visible on the monitor displaying the electronic image or on the hard copy depends on the resolution of these display devices, the required size of the displayed image and the scaling required to achieve that size.
Another source of artefacts is irreversible image compression and expansion, such as JPEG. The term irreversible stands for the fact that the second electronic image obtained by expansion of a compressed version of a first electronic image is not guaranteed to be identical to the first electronic image. According to a JPEG technique, the first electronic image is partitioned in sections of 8xc3x978 pixels. Each section is compressed to a data format that occupies less memory than the memory required to store the 8xc3x978 or 64 pixels. The thus compressed image may be temporarily stored in a memory means or transmitted to another image processing system. Whenever the image is needed for display on a monitor or for reproduction, the compressed data format is expanded to yield sections of 8xc3x978 pixels. Assembly of these expanded 8xc3x978 sections gives a second electronic image, very similar to the first electronic image. According to the JPEG compression technique, a compression ratio may be specified. The higher the compression ratio, the more the second electronic image may be different from the first electronic image. This difference may be apparent in the second expanded electronic image, even without visually referring to the first original electronic image. The 8xc3x978 sections may be identified by the naked eye. Also this effect is unwanted and may need correction.
Electronic images may already suffer from the staircasing effect, either because an electronic scaling already took place or because the electronic image has undergone a JPEG compression and expansion. Electronic scaling or zooming such image will further deteriorate the quality of the image, possibly by introducing more staircasing effects.
According to prior art methods, the user may interactively adapt the pixel values in areas where the staircasing effects are most objectionable. But this is a tedious work. There is thus a need to assist the user by electronic image processing means to remove these unwanted effects.
It is an object of the invention to provide a method for attenuating or removing the effect of staircasing when low resolution images are electronically enlarged.
It is another object of the invention to enlarge electronic images such that the image quality is comparable to an electronic image acquired at a higher resolution.
The above mentioned objects are realised by a method having the specific features defined in claim 1. Specific features for preferred embodiments of the invention are set out in the dependent claims.
The method according to the current invention proceeds in the following way. Preferably, each pixel of the electronic image is selected for examination. Preferably, this is done in an orderly way, e.g. starting from the first pixel in the first line, proceeding to the second pixel in the first line, etc. For each pixel, at least two different orientations are defined, e.g. vertical and horizontal. Preferably, four orientations are defined, including horizontal, vertical and two oblique directions at +45xc2x0 and xe2x88x9245xc2x0 with respect to the vertical direction. Also three, five, . . . directions may be defined. Each direction may be materialised by a straight line running through the selected pixel. For each direction a gradient value may be computed. In its theoretical definition, the gradient value is the amount of change of the pixel value for a unit displacement according to the chosen direction. In the detailed description various implementations will be given for the computation of the gradient value, suitable for application to a digital electronic image. Each direction may give a different gradient value. Therefore, with each direction a gradient value for the selected pixel is associated. Since with each direction a line is associated, each gradient value is also associated with a line.
According to the current invention, the gradient values according to different orientations or directions are decisive for the operations to be done on the selected pixel value. Therefore, one of the plurality of directions or defined lines is selected based on the value of the associated gradient. In one embodiment, the direction having the largest gradient value may be selected. According to another embodiment, the direction having the smallest gradient value may be selected. The orientation of the selected direction is decisive for the orientation of a second direction along which smoothing may be done. The second direction corresponds to a second line, which may be positioned to run through the selected pixel. This second direction has a specific orientation with respect to the first selected direction. In a preferred embodiment, the second line has an orientation orthogonal to the orientation of the first line having the largest gradient value. In another embodiment, the second line has an orientation parallel to the orientation of the first line having the smallest gradient value. In both cases, the second line has a specific orientation relative to the selected first line.
The values of at least two pixels situated on the second line, will be decisive for the new pixel value of the selected pixel.
According to a preferred embodiment, the gradient value may be computed based on at least two pixel values: the pixel value of a first pixel P1 and the pixel value of a second pixel P2. Both P1 and P2 are situated on the defined line, associated with the gradient value. Thus the gradient G is a function of P1 and P2, i.e. G=G (P1,P2). The function G( ) is a function having two arguments. G( ) is selected such that if the value of P1 increases, the value of G does not decrease. This means that the value of G may remain constant or increase when P1 increases. If the value of P2 increases, the value of G does not increase, i.e. G remains constant or decreases when P2 increases.
According to a specific embodiment, G=P1xe2x88x92P2. More generally, G=W1.P1xe2x88x92W2.P2. W1 and W2 are two constant positive weight factors, for multiplication with P1 and P2 respectively. The value of W1 and W2 may depend on the location of P1 and P2 relatively to the selected pixel. According to a more specific embodiment as discussed in the detailed description, G is the difference of two (weighted) sums of pixel values. The first sum includes pixels situated on a line orthogonal to the defined line. The second sum includes pixels located on a line parallel to that orthogonal line, or pixels situated on a lines parallel to the direction of the defined line and including the pixels from the first sum. A weighted sum means that each pixel value may be multiplied with a positive weight factor, before it is accumulated to the sum.
If the largest gradient value is taken to decide on the direction along which the smoothing will be performed, then it is preferred that the second line used for computation of the new pixel value is oriented substantially orthogonal with respect to the orientation associated with the largest gradient value. A second line is substantially orthogonal to a first line selected from a plurality of N lines having different orientations, if the angle between the first and second line is in the range of [90xc2x0xe2x88x92xcex94,90xc2x0+xcex94], where xcex94=90xc2x0/N. If for example the gradient is examined according to four directions, then N=4. The angle between the first and second line may then be from 67.5xc2x0 to 112.5xc2x0 in order to be substantially orthogonal. The orthogonality must be measured on the reproduced visible image, because it is possible that the spatial resolution of the electronic image is non-isotropic, i.e. that the distance between the centres of two neighbouring pixels in the horizontal (line) direction is different from such distance in the vertical direction. Such situation is also referred to as an electronic image having rectangular pixels.
According to a preferred embodiment, the new pixel value is computed from at least two pixels situated on the second line, having the direction according to which smoothing must occur. These two pixels are referred to as third and fourth pixel P3 and P4 respectively, whereas the third or fourth pixel may well coincide with the aforementioned first or second pixel. In order to achieve a smoothing operation, it is advantageous to compute the new pixel value based on a smoothing function S=S (P3,P4). The smoothing function has the property that as one of its arguments P3 or P4 increases, the function value either remains constant or increases; i.e. S is a non-descending function of its arguments. According to one embodiment, S includes a weighted sum of P3 and P4, e.g. S (P3,P4)=W3.P3+W4.P4.
According to a more preferred embodiment, the corrected image is further processed for sharpening, also based on gradient calculations.